V-Optimal Filters for Data Approximation in Continuous Data Streams

Monitoring data streams in real time over distributed streaming environments plays a large role in maintaining situation awareness, which is a great challenge due to huge data volumes and bandwidth limitations. In this monitoring process, transmission cost and value accuracy are two very important but conflicting factors in measuring the efficacy of the system. On one hand, increasing value accuracy increases transmission cost. On the other hand, reducing transmission cost, which can be accomplished by smoothing the data, will reduce value accuracy. In this paper, we use V-Optimal histograms to approximate the data distribution at the data sources. The V-Optimal algorithm is used for computing optimal number of buckets and bucket boundaries given a certain error bound, which are then used to approximate and communicate data between the source and the server. We introduce the notion of a soft precision constraint (PC) and use two additional metrics namely, weighted average error and PC violation rate or error rate to control the quality of approximation. We show through extensive experimentation that our approach performs very well in maintaining data quality as well as reducing communication cost.